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ABSTRACT 

Portfolios are increasingly used in postgraduate medical education and in gastroenterology training as an assessment 
tool, as documentation of competence, a database of procedure experience (for example endoscopy experience) and for 
revalidation purposes. In this paper the educational theory behind their use is described and the evidence for their use is 

discussed. 
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Introduction 

The use of portfolios for assessment in the 
health care professions has developed as part of 
a move away from "snap shot" examinations, 
towards broader methods of assessment. This 
form of assessment is thought to encourage 
closer links between assessment and learning by 
improving learning outcomes using assessment 
and providing feedback. The use of a portfolio 
is also thought to enhance the assessment of 
areas that are difficult to assess by traditional 
methods in a wide range of clinical contexts. 
These areas include attitudes, personal 
attributes, reflection and professionalism (1). 
Portfolios are being increasingly used in 
postgraduate medical education to assess ability to 
progress. In the UK, a huge emphasis is placed 
upon them in the foundation years of training (2) 
and later in speciality training (3). They also form 
the basis of the annual appraisal (ARCP) for 
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trainee doctors in the UK (2,3). They are used 
extensively in training in Gastroenterology as a 
way of documenting clinical competence and for 
recording endoscopy experience and competence 
(4). In 2001, the general medical council (5) 
identified a portfolio approach to revalidation for 
all doctors. Revalidation was introduced in the 
United Kingdom in 2012 and requires licenced 
doctors to keep a portfolio of evidence of their 
practice (6). 

Portfolio contents 

Several models of portfolios have been 
described. In Table 1, their use is described and 
the advantages and disadvantage of their use is 
outlined. The four models demonstrate how 
different the contents of a portfolio may be. The 
appropriateness of each model depends very much 
on its purpose and whether it will be used in an 
assessment process. (7). 

Alignment and assessment 

In order to demonstrate alignment, a portfolio 
structure should be decided based on the format 
the evidence is required in. The more prescriptive 
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Table 1. Models of portfolio 



Portfolio 



Description 



Advantages 



Disadvantages 



Shopping Contains anything which has been 
trolley produced or used during the 

learning process 
Toast rack "Toast" for each period of learning 



Cake mix Integration of the parts 

"Mixing" is reflection on the 

analytical components 
Spinal Series of competency statements are 

column the "vertebrae" 

Evidence is linked to the 

competencies 



Very inclusive 



Corresponds with the 
curriculum 
Can be marked 
Includes reflection 
Global assessment 



Each competency has 
its own evidence 



Difficult to assess. No analysis 
of contents 

Each item is discrete and does 
not provide overall assessment 
of learning 
No overall reflection 
Individual components may not 
be clear 



portfolio models should be easily aligned with the 
curriculum as the curriculum can form the "spinal 
column" or "toast rack". The "shopping trolley" 
model is least likely to demonstrate alignment as 
this method relies heavily on the student including 
relevant documentation. Whichever method is 
chosen should provide ample opportunity to 
demonstrate learning in many different ways and 
should be a holistic record of learning. 
Davis and Ponnamperuma (2006) propose five 
steps in the portfolio assessment process. They are 
as follows: 

1 . Collection of the evidence of learning. 

2. Reflection of the learning. 

3. Evaluation of evidence by assessors. 

4. Defence of evidence by the individual 
being assessed. 

5. An assessment decision. All the assessors 
involved, should make a decision based on 
pre-defined criteria. (1) 

Educational impact 

The different types of portfolio will have 
varying degrees of educational impact. A model, 
which purely records educational events but does 
not provide overall reflection (for example the 
"toast rack" may not foster learning and may be 
seen as a box ticking exercise. In contrast, the 
"cake mix" model that includes evidence and 



reflection may increase educational impact by 
enabling an individual to reflect on learning. 

An advantage of using portfolios as a means of 
assessment is that gaps may be identified in 
training particularly if the "spinal column" or 
"toast rack" models are used. It is also possible to 
present evidence from other assessments within a 
portfolio. For example, in postgraduate medical 
education work place based assessments are used. 
Incorporating these elements will allow the 
assessment process to be greater than the sum of 
its parts. 

Reliability 

Roberts et al. (2002) conducted a systematic 
review of the evidence of the use of portfolios in 
the medical education literature (9). Two papers 
were found which provided data from small 
studies (10, 11). These studies focused on the 
inter-rater reliability of portfolios. Their results 
showed reliability falling below the level of 0.8 
(9). Jenkins et al., (2013) tested the reliability of a 
portfolio used as an assessment tool for 
postgraduate family medicine training in South 
Africa and found the tool as a whole to be reliable 
with a co-efficient of 0.92. Individual components 
of the portfolio were not shown to be reliable, 
which lead the researchers to make 
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recommendations as to how the portfoUo could be 
improved (12). 

The literature suggests that in order to achieve 
high levels of inter-rater reliability, portfolios 
should be introduced to well prepared students and 
should be of uniform content (13, 14). They 
should be marked by well trained, experienced 
assessors using clear criteria. Both assessors and 
assesses should have a shared understanding of the 
purpose of the assessment (9). 

Portfolios may include other assessments 
within them for example in post-graduate 
education, work-based assessments such as DOPS 
(directly observed procedural skills), CBD (case 
based discussion) and multi-source feedback. 
Generalizability studies have shown that content 
specificity is a major contributor to unreliability 
(15). This means that large samples of 
performance need to be tested before a reliable 
judgement about ability can be made (9). This 
means that by including work based assessments 
in a portfolio, reliability is increased, as there will 
be more assessments. This allows individual 
competencies to be assessed in more than one 
way, and enables triangulation of evidence and 
therefore increases reliability. 

Validity 

Dreissen et al. (2006) identified a few studies 
in the literature, which lend some support to 
different aspects of validity. These include 
criterion and construct validity, predictive validity 
and content validity. (16-19). 

Driessen (2006) looked in detail at which 
criteria affected an assessor's judgement of 
student' s reflective skills and found that there was 
no impact of irrelevant criteria (such as hand 
writing) in assessment, and that the strongest 
predictor of variance was the "quality of 
reflection "(19). Driessen et al., (2006) suggested 
that portfolio standardisation and the use of 
analytical criteria with the aim of improving 
reliability may threaten validity by limiting room 



for the description of personal learning 
experiences in different authentic situations (19). 
They also suggested that including checklists and 
analytical assessment criteria might trivialise the 
assessment process (19). A study by Quinlan 
(2002), it was demonstrated that competence 
ratings were influenced by what assessors already 
knew about students, and therefore reducing 
validity of the assessment tool (20). 

The use of portfolios in medical training to 
demonstrate clinical competence 

Portfolios were initially used in medical 
education for formative purposes in order to 
encourage reflection. (21). More recently, the use 
of portfolios has been advocated for summative 
purposes (9, 22). The "eportfolio" now forms the 
basis for both foundation year training and 
speciality training in the United Kingdom. This is 
a spinal column type of portfolio and individual 
pieces of evidence are linked to predefined 
competencies (2, 3). For endoscopy training in the 
UK trainees are expected to document all their 
procedure data in one database, and to collect 
work based assessments during their training. A 
trainee is deemed competent when they have 
achieved certain predetermined criteria laid out in 
the eportfolio. This documentation is available for 
trainers to explore (3). 

Table 2. Components of Millar's pyramid (23) 



Millar's pyramid 



Element of the portfolio 



Does 


Components of a portfolio: 




work based assessments, 




reflective diaries, 360 




degree appraisal 


Shows how 


Clinical examinations 


Knows how 


Some written examinations, 




e.g. clinical examinations. 


Knows 


Written Examinations 



Millers pyramid 

Millar's pyramid is often used as a model for 
the assessment of clinical competence. The 
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pyramid has four tiers that represent does, shows 
how, knows how and knows (23). Table 2 
demonstrates how each aspect of Millar's pyramid 
may be demonstrated using a portfolio. 

Portfolios are advantageous in that they are 
able to test areas difficult to assess such as 
professionalism, continuous professional 
development, attitudes and critical thinking. They 
may be thought of as encouraging more effective 
learning by encouraging reflection and allowing 
continuous assessment to occur. Work based 
assessments provide real Hfe assessments of real 
life situations and by including different methods 
of assessment, an individual is assessed in many 
different ways. Problems should be able to be 
identified early and gaps in an individual's 
learning may be identified. Feedback can be 
provided following individual assessments and 
after reviewing the entire document. If structured 
appropriately it should be feasible and acceptable 
to be used in summative assessments (1). 

However, the environment in which the 
portfolio is introduced may influence attitudes 
towards portfolios. There may also issues around 
the practical nature and feasibility of portfolio 
assessments. They are time consuming for both 
the assessor and the individual being assessed and 
they are difficult to difficult to mark (1). They 
require evidence, assessments and data to be 
collected prospectively and cannot be effectively 
competed retrospectively (1). 

Conclusions 

Portfolios may be used to teach and assess 
attitudes and professionalism that are difficult to 
assess by other means. They may be used to teach 
and learn a range of outcomes. If they incorporate 
assessments from a range of settings they should 
be able to assess all four levels of Millar's 
pyramid if results from work based assessments. 
They may provide feedback to both the individual 
being assessed and the assessor, and a holistic 



picture of the individual's fitness to practice (1). 
They are widely used in medical education to 
demonstrate competencies across a range of 
clinical and non-clinical skills and this use will 
improve the reUability and vaUdity of the tools 
used. There are some disadvantages, which have 
also been described which make using a portfolio 
as a means of assessment difficult, but if 
completed appropriately they provide good 
evidence for summative assessments and for 
revaHdation purposes. 
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